Music Motion

Wiring i2c Connections

We used circuit python libraries compatible with our accelerometers (see references) and used i2c as our communication protocol. The i2c required 4 connections: Pi 3V to sensor VCC (red wire), Pi GND to sensor GND (black wire), Pi SCL to sensor SCL (yellow wire), Pi SDA to sensor SDA (blue wire).

Since we were using two accelerometers, we used i2c0 and i2c1 pins on our raspberry pi. It was important to use these ports (ports 1 and 0) because other i2c ports associated with the raspberry pi 4 are not compatible with the circuit python library, due to the raspberry pi 4 being a newer model. Therefore, we used SCL1 (GPIO3), SDA1 (GPIO2), SCL0 (GPIO1), and SDA0 (GPIO0).

There was no formal testing for this step besides watching the accelerometer light turn green for “on”, and verifying that our wiring matched our diagrams on both the RPi4 pinout side and on the accelerometer side. More testing came in the next section when we verified our i2c connections.

Close Window

Setup and Configuration

We first followed the steps for circuit python outlined in our reference section. This included adding libraries adafruit_lsm6ds.mpy, adafruit_bus_device, and adafruit_register. Please refer to that website reference to see the specific steps taken for adding these libraries.

In order for i2c to work on port zero, some setup and configuring needed to be done on the raspberry pi. We followed the steps outline in our i2c reference, listed in references. Please refer to that website reference to see the specific steps taken for i2c setup.

However, we made a few small changes to the i2c directions. We did not update our raspberry pi, as we needed the Wheezy downgrade for pygame. Additionally, we added the following lines of code to /boot/config.txt because the directions do not include how to enable i2c0. This code was found from several raspberrypi.org forums.

# added for i2c

dtparam=i2c0=on dtoverlay=i2c-gpio,bus=0,i2c_gpio_delay_us=1,i2c_gpio_sda=0,i2c_gpio_scl=1

dtparam=i2c1=on dtoverlay=i2c-gpio,bus=1,i2c_gpio_delay_us=1,i2c_gpio_sda=2,i2c_gpio_scl=3

To test we had setup everything correctly, we ran the command ls /dev/i2c* in the terminal. The output was /dev/i2c-1

This meant we had already configured the raspberry pi for i2c1. After configuring the raspberry pi for both i2c1 and i2c0, we rebooted.

After rebooting, we ran the command again and got the output:

/dev/i2c-0 /dev/i2c-1

This output was what we were expecting, and it meant our configuration step had worked to set up both i2c0 and i2c1.

Close Window

Testing Wiring and Configuration Steps

To test these i2c ports were connected to the accelerometers, we ran command i2cdetect -y 0 for i2c port 0 and i2cdetect -y 1 for i2c port 1. We first tested port 1. At first, these readings were all dashes, meaning the accelerometer was failing to write to a i2c address. The problem was that the pins are not soldered, and therefore the metal connection is very weak. After pressing the accelerometers flush against the metal pins, i2cdetect registered the accelerometer at address 6a.

We were at first concerned when seeing that address 50 was already in use for i2c0. But after conducting research on i2c, we realized that it is a feature of i2c to run multiple i2c devices on the same channel, so long as they are writing to different addresses. For that reason, we could use i2c0 and i2c1 for our accelerometers. We could not, however, put both accelerometers on the same channel (otherwise referred to as the i2c port) because they write to the same address, 6a.

Close Window

Gather Accelerometer Data

Once the i2cdetect command was yielding positive results, and we determined the accelerometers were wired correctly, testing the data collection was very simple. The library provides example library code for how to print acceleration data to the terminal every half second. This code, see accel_example.py, worked well, but it was difficult to determine proper thresholds purely by reading long strings of data every half second on the console. We decided we would need to write more code to determine good thresholds.

Close Window

Thresholds for x (up/down) and z (left/right) Directions

We created a new data collection script, accel_graph.py that would graph the accelerometer data, making it easier to determine thresholds. This was more useful because it was easier to read and because we were taking data continuously rather than pausing for a half second. This graph was very useful, and is featured in our report appendix. For this graph, Sabrina moved the accelerometer down, up, left, right, forward, and then backward. These directions correspond to +x, -x, +z, -z, +y, and -y respectively.

In the diagram, you will notice movements in the opposite direction in addition to the intended direction, looking similar to a sine wave. This is because it is natural human movement to “wind up” or “reel back” when waving the accelerometer. The “reel back” phenomenon is what is causing the sine wave. Sabrina was careful to avoid the “wind up” phenomenon when creating this data.

We decided to implement different thresholds for left and right accelerometers because it is easiest to hold the accelerometers when they mirror each other, thereby inverting the x and z axis. For example, while negative x means “up” for one accelerometer, it means “down” for the other.

Further adjustments were made in accel_thresholds.py. In that code, data is averaged to determine first true movement, eliminating the “wind up” problem discussed earlier. Additionally, there is a pause_time variable and ready_move() function that disable data collection for a short period of time after determining a move has occurred. This disable step removes the “reel back” problem discussed earlier. However, the reel back problem didn’t end up translating to the game. Once the correct movement is detected, the game waits for the next movement to start data collection again anyway.

Close Window

Phone accelerometers

First we explored the option of using phone accelerometers via an app. We determined this to not be a viable solution for many reasons. Firstly, we had different phones: one was an iphone and the other was an android. Secondly, the app phyphox does not provide an easy way to stream bluetooth data onto another device. Lastly, the raspberry pi was not pairing with the phone. The phone would consistently request the user “forget the device” because of the failure to connect properly over bluetooth.

Close Window

Home Screen

The home screen functions very similar to that of the one in control_two_collide.py. Text is displayed on the screen. From the home screen, a player can change the song, start, or quit. The song changes by changing the position of the index song in the song list (described in Music) and updates the screen with the selected song. The start button draws the play screen and starts playing the music and moves. The quit button exits out of the main while loop entirely and ends the game_screen.py run. This was tested by initially adding some placeholder songs and pressing different areas of the screen to make sure the intended action followed.

Close Window

Play Screen

The play screen shows the current move for both hands as well as the next two moves per hand. The most difficult aspect of the game screen design was creating the arrows and highlighting the correct arrows for a move. We looked online for tutorials for drawing arrows in PyGame. We also created a helper function in constants.py to generate the end and start positions based on the center position, size, and direction of the arrow since the center position was easier to use for placing the arrows symmetrically on the screen. To update the main arrows, we checked the direction of the arrow against the direction of the current move before incrementing the move based on the bpm on the song. For the queue arrows, we reused the draw arrow function but scaled the arrows down. To update the queue arrows, it would draw the next and next next movements from the song object. We tested this by running a song with a couple moves added to it and seeing if it matched the screen. After that, we incorporated the accelerometers. We got the data for the left and right hand, determining the users move and compared that against the song's choreographed move. We incremented the score based on whether these values matched.

Close Window

End Screen

The final score screen was the easiest screen. After the song is finished, the screen is cleared and then the score information is added on the screen. And the "Home" button from the play screen is carried over so the user can restart another song. This was tested after integration, so we printed the current score after each move throughout the song and checked that the total score displayed matched at this ending screen.

Close Window

Movement.py

In our movement class, we defined what constitutes a movement object and gave functions to help determine what the game screen should display in terms of active arrows. A movement is a move set that includes the information about what the left and right motion should be at a single time. A pattern consistent throughout the code was associating ints with directions: -1 for no direction (blank), 0 for up, 1 for left, 2 for right, and 3 for left. We also added some methods determining what color an arrow should be to help color active arrows in the game screen. Overall, this class is very simple. It can be expanded on in the future if we want to associate more things with a given move. For example, we could add a score attribute and give each move a different score based on their difficulty. Due to the simplicity of this class, we did not write or perform tests specifically targeted for it.

Attributes and methods of the class consist of:

Close Window

Song.py

In our song class, we define useful functions to utilize by our game running in game_screen.py. The song has important information relating to the song and keeps track of where in the song the moves should be. First, a song is created and then moves can be added with methods as seen in the songs list. Then it is played and incremented in the game screen before being reset when completed or exited out of.

Attributes and methods of the class consist of:

Close Window

Songslist.py

In our songslist file, each song is associated with sequences of “dance moves”. Examples of song sequences are the chorus, the verse, and the outro. The dance moves are four left and four right hand movements, for example: left up and right up, left down and right down, left up and right up, and left down and right down. First, a song object is created and instantiated. Then, move sequences in the form of lists with left and right values for a given movement are added to the song. Each song object is then added to a songs list. The game screen accesses this to play the song.

An idea pitched to us by one of our TAs was to do processing on our song to auto-generate dance moves based on notes, volume, etc. While we found this to be a very interesting idea, we ultimately decided it would not suit our needs for this project for two main reasons.

The first was because there is no need to do real time music processing during the game. The processing instead would occur once before adding the song as an option for the user. If the moves for a song are only generated once, there is no need to spend hours developing music analysis code. Instead, creating a library of dance moves that can be reused both in this song and for future songs seemed like the most practical solution.

The second reason is that auto-generated movements don’t take into account usability. We designed moves, like the one in the picture above, that are very intuitive for the user. These moves are repetitive, interesting, and easy enough that left and right hand movements can be done at the same time.

Close Window

BPM Detection

We ran music analysis on our song to determine the bpm - beats per minute. This is very important for timing the music to the movements, as we change the dance movement every other beat of the music. We did not create our own code for this, but utilized a python script found online. The bpm for senorita was determined to be, on average, 120bpm. This is in contrast to sources online that we saw that said 117bpm. We tested both and found 120 to be better suited to match the beat. Therefore, we change the movement 120/2 beats per minute, conveniently, once every second, since 120 bpm was too fast. Please see the references section for more information about the bpm script used.

Close Window

Testing

Since the music libraries were created last and were the simplest part of the project, they were tested by integrating them directly into our game. The main way to test our song was to run the game repeatedly, listening and watching for correct results. We watched the movements indicated on the screen, comparing them to the moves for that part of the song. We determined that they indeed lined up correctly with what we had written.

However, there were small timing inconsistencies. In a song, it is common for an artist to speed up or slow down the tempo for a small part of the song. Our solution was to create a move called “Move 1” that was 1 movement long instead of 4 movements long. We then incorporated this move just after verse 1 and again at the end of the song. This fixed our timing issue.

Close Window